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A  Sampling  Procedure  and  Public  Policy 
G.  J.  Lieberman,  I.  Olkin,  F.  Riddle 
Stanford  University 

Following  the  March,  1979  publication  of  the  revised  Office  of  Manage¬ 
ment  Budget  Circular  A-21  "Cost  Principle  for  Higher  Education",  which 
promulgates  the  cost  principles  to  be  used  in  the  determination  of  cost 
charged  to  government  sponsored  agreements,  there  was  an  undercurrent  of 
faculty  dissatisfaction  with  the  revised  rules.  Of  major  issue  was,  and 
still  is,  the  requirements  relating  to  reporting  of  faculty  effort  charged 
directly  or  indirectly  to  government  agreements. 

The  two  acceptable  methods  for  effort  reporting  contained  in  the 
revised  A-21  are  contained  in  section  J.6.  of  that  document  and  are  called 
Monitored  Workload  and  Personal  Activity  Reporting.  The  Monitored  Workload 
method  can  be  used  for  Faculty  and  Professionals  only,  whereas  the  Personal 
Activity  Reporting  method  can  be  used  for  all  personnel. 

The  major  objection  raised  by  the  faculty  centered  on  the  requirement 
(under  either  system)  to  account  for  100%  of  their  effort  applicable  to  each 
sponsored  agreement,  each  indirect  cost  category,  and  each  major  function 
of  the  University.  This  level  of  accountability  was  considered  burdensome, 
arbitrary,  and  to  some  extent,  useless  work  because  of  the  Interrelated 
nature  of  the  research,  instruction  and  other  activities  of  the  faculty. 

The  results  of  such  effort  reports,  though  fulfilling  the  documentation 
requirements  established  by  the  regulation,  were  known  to  be  arbitrary  and 
therefore  inaccurate,  and  with  the  added  detail  terribly  expensive  for 
institutions  to  Implement. 
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That  che  assessment  of  direct  and  Indirect  costs  is,  at  times,  highly 
subjective  and  approximate  was  also  recognized  by  the  government.  Section 
J.6.b  of  OMBA-21  states: 

.  .  it  is  recognized  that,  because  of  the  nature  of  the 
vork  involved  in  academic  institutions,  the  various  and 
often  interrelated  activities  of  professional  and  profes¬ 
sional  employees  frequently  cannot  be  measured  with  a  high 
degree  of  precision,  that  reliance  must  be  placed  on 
reasonably  accurate  approximations  and  that  acceptance  of 
a  degree  of  tolerance  in  measurement  is  appropriate." 

Because  of  dissatisfaction  with  the  new  effort  reporting  guidelines, 
the  major  research  institutions  formed  a  task  force  under  the  auspices  of 
the  Association  of  American  Universities  (AAU)  to  develop  alternative  methods 
of  gathering  faculty  effort  without  the  burden  necessitated  by  the  new 
regulations.  As  a  consequence  of  their  deliberations,  it  was  suggested  that 
a  method  of  statistical  sampling  be  used  to  determine  the  indirect  effort 
performed  by  the  faculty.  The  results  of  the  sample  could  then  be  used  in 
determining  indirect  cost  rates. 

It  was  also  noted  that  too  often  the  approximations  made  in  effort 
reporting  resulted  in  very  poor  data.  To  improve  the  data,  the  faculty  would 
have  to  be  apprised  of  what  specific  costs  are  to  be  labeled  a  direct  or 
indirect  cost.  By  cutting  the  scope  of  reporting,  a  sampling  procedure  would 
permit  increased  dialogue  between  the  administration  and  faculty  thereby 
enabling  the  faculty  to  have  a  better  understanding  of  the  process. 
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The  idea  of  statistical  sampling  was  explored  between  Stanford  Univer¬ 
sity  and  the  Office  of  Management  and  Budget.  The  OMB  felt  the  idea  had 
possibilities  and  suggested  that  Stanford  submit  a  formal  proposal.  The 
authors  carried  out  some  preliminary  work  to  describe  how  such  a  system 
would  work,  and  a  plan  was  presented  to  (SIB  and  a  panel  of  Federal  statis¬ 
ticians.  After  their  review,  agreement  was  reached  to  proceed  with  a  more 
detailed  study  to  include  the  process,  the  sample  sizes  to  reach  a  90% 
confidence  level  with  a  probable  5%  error  and  the  statistical  formulas  to 
produce  the  desired  data. 

The  plan  was  completed  in  January,  1981  and  submitted  and  accepted  by 
OMB  which  had  chosen  ten  universities  who  had  expressed  Interest  in  using 
the  statistical  method  as  test  of  the  proposal.  Unfortunately,  of  those 
institutions  who  submitted  their  proposals  based  on  the  statistical  model, 
the  implementing  agency  wanted  the  test  made  in  parallel  with  a  standard 
J.6.  method  so  that  the  results  could  be  verified.  In  essence,  this  stringent 
requirement  virtually  terminated  further  consideration  by  the  institutions. 

However,  on  January  7,  1982,  after  several  months  of  added  effort  by 
the  AAU  and  other  higher  education  associations,  a  proposal  for  revision  of 
section  J.6.  of  A-21  was  placed  in  the  Federal  Register.  Included  in  the 
proposed  revision  is  paragraph  J.6.b(2)(c),  which  reads: 

(c)  The  payroll  distribution  system  will  allow  confirmation  of 
activity  allocable  to  each  sponsored  agreement  and  each 
category  of  activity  needed  to  identify  Indirect  costs  and 
the  functions  to  which  they  are  applicable.  The  activities 
chargeable  to  Indirect  cost  categories  or  the  major  functions 
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of  Che  institution  for  employees  whose  salaries  must  be 
apportioned  (see  J.6.b(l)(b)  above)  if  not  initially 
Identified  as  separate  categories,  may  be  subsequently 
distributed  by  any  reasonable  method  mutually  agreed  to, 
including,  but  not  limited  to,  suitably  conducted 
surveys,  statistical  sampling  (emphasis  added)  procedures, 
or  the  application  of  negotiated  fixed  rates. 

It  is  assumed  that  the  public  comment  will  be  overwhelmingly  favorable 
to  the  proposed  revision,  which  makes  the  publication  of  the  method  developed 
at  Stanford  by  the  authors  timely.  Use  of  the  method  would  allow  confirma¬ 
tion  of  salaries  directly  charged  to  sponsored  agreements  by  confirming 
notations  on  the  monthly  expenditure  reports  and  less  costly  (and  less 
disruptive  for  the  faculty),  gathering  of  data  necessary  for  indirect  costing 
purposes.  What  follows  is  the  proposal  that  was  submitted  to  OMB  as  a 
statistical  sampling  model. 
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2.  The  Sampling  Plan 

The  proposed  sampling  plan  is  designed  for  a  particular  university 
for  a  particular  purpose.  However,  the  method  can  be  extended  to  other 
organizations  and  for  other  purposes. 

The  sample  is  designed  to  determine  the  amount  of  effort  expended 
by  the  faculty  in  indirect  activities  such  as  departmental  Research, 
university  and  student  service  administration.  For  example,  if  the  indirect 
activity  departmental  administration  is  of  interest,  then  we  need  to 
estimate  the  total  cost,  T,  expended  by  the  faculty  engaged  in  that 
activity: 

T  »  flSl  +  fNsN  , 

where  f^  is  the  fraction  of  time  that  the  j  — th  faculty  members  devotes  to 
departmental  administration,  s^  is  the  academic  year  salary  of  the  j-th 
faculty  member  and  N  is  the  number  of  relevant  faculty.  There  will  be 
similar  totals  for  university  administration,  research  administration,  etc. 

Because  faculty  differ  with  respect  to  their  activities,  we  use  a 
stratified  sample  thereby  reducing  the  sample  variance  of  our  estimates. 

Two  methods  of  stratification  appear  natural  in  a  university  context,  namely, 
rank  and  disciplines. 

For  stratification  by  rank  four  strata  may  be  appropriate:  professors, 
associate  professors,  assistant  professors,  department  chairs  and  faculty 
with  major  administrative  duties.  Stratification  by  discipline  might  include 
the  Schools  of  Medicine,  Engineering,  and  Education,  the  Physical  Sciences, 
Social  Sciences,  the  Humanities,  etc.  In  making  these  divisions  it  is 
important  to  emphasize  that  within  each  stratum  the  faculty  should  be  relatively 
homogeneous. 
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Definition  of  the  Universe 

The  universe  is  assumed  to  be  all  faculty  who  conduct  sponsored 
activities  and  have  other  duties  funded  from  non-government  sources. 
Including  the  departmental  operating  budget.  Also  included  would  be  faculty 
totally  funded  from  the  operating  budget  or  a  combination  of  operating 
budget  and  other  non-government  sources.  A  faculty  person  funded  100Z 
on  sponsored  agreements  or  patient  care  would  be  excluded  from  the  universe. 
Since  the  results  of  the  sample  needs  to  be  generalized  to  the  universe 
it  represents,  the  sample  must  reflect  the  elements  necessary  to  fulfill 
the  objective  of  the  universe. 

The  Strata 

At  Stanford  University  strata  were  chosen  using  a  combination  of 
both  the  rank  and  discipline  criteria.  These  were: 

1)  Medical  School  department  chairs. 

2)  Other  department  chairs  and  laboratory  directors. 

3)  Full  professors  in  the  sciences. 

4)  Full  professors  in  other  fields. 

5)  Associate  professors. 

6)  Assistant  professors. 

3.  Details  of  the  Sampling  Plan 

3.1  Preliminaries .  For  each  stratum,  we  have  available  the  total 
number,  N^,  of  faculty  members,  and  the  average,  f^,  of  the  product 
of  academic  year  salary  and  proportion  of  time  devoted  to  departmental 
administration.  Consequently,  we  can  also  determine  the  sample  variance, 
S*.  of  the  Y-values.  That  is 


S*  -  I(Y-Yi)2/(Ni-l)  . 
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Actually,  much  of  the  above  data  is  unknown  beforehand.  However , 
some  preliminary  estimates  are  essential  if  we  are  to  sample  with  a 
specified  degree  of  accuracy.  One  procedure  is  to  obtain  estimates  from 
past  data,  or  from  a  sample  of  past  data.  Indeed,  an  examination  of  past 
data  may  also  lead  to  improved  stratification.  At  Stanford  an  examination 
of  1979-1980  data  was  used  to  estimate  the  needed  parameters  and  also  to 
establish  the  use  of  the  strata. 

3.2  Choice  of  Sample  Sizes.  The  design  depends  on  choosing  an  overall 
sample  size,  n,  and  an  allocation  into  the  various  strata.  The  total 
sample  size  depends  on  the  confidence  level,  c,  and  a  given  relative 
error,  e.  The  higher  the  required  confidence,  the  larger  the  required 
sample  size;  the  smaller  the  relative  error,  or  equivalently,  the  greater 
the  precision,  the  larger  the  required  sample  size.  The  sample  size  also 
depends  on  the  variances  in  each  stratun,  but  not  in  a  simple  way. 

The  determination  of  n  is  given  by 


(ENiSi)^ 
W2  +  Et^S2 


T 

V2  +  V 


where  T  ■  IN^S^,  V  ■  IN^S2  and  W  depends  on  the  confidence  level,  c, 


and  the  relative  error  e: 


W  -  e (Total) /c* 


(2) 
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where  the  total  is  the  estimated  true  total  cost  of  administration, 
c*  ■  0  ^ ((l+c)/2),  and  ♦  is  the  standard  normal  cumulative  function. 

Once  the  sample  size  is  chosen,  the  allocation  n^  to  stratum  i 
is  made  in  the  proportion 


,  11, 

ni  "  'nrs*  * 


rj 


»i  <  »i 


(3) 


3.3  An  Example 

The  following  data  with  5  strata  are  fictitious. 


Stratum 

Number 

in  stratum 

2 

Variance  S. 
in  each 
stratum 

Stratum 

standard 

deviation 

NiSi 

NiSi 

ZN.S. 
j  3 

Kisi 

(in  millions) 

(in  thousands) 

(thousands) 

(in  millions) 

A 

B 

C 

D 

E 

F 

G 

1 

20 

100.00 

10.0 

200 

.110 

2000 

2 

50 

4.00 

2.0 

100 

.055 

200 

3 

200 

9.00 

3.0 

600 

.329 

1800 

4 

250 

6.25 

2.5 

625 

.342 

1562.5 

5 

300 

1.00 

1.0 

300 

.164 

3.0 

T- 1,825 

1.000 

V- 5,862.5 

If  we  wish  to  estimate  the  total  amount  spent  on  departmental  administration 
to  within  5Z  of  the  true  total  with  90%  confidence,  and  if  we  estimate  the 
true  total  to  be  more  than  $7  million,  then  from  (2) 


W  - 


1 


.000.000) 


1.645 


212,766 
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where  c*  -  1.645  is  obtained  from  the  tables  of  normal  distribution. 
From  (1) 


n  > 


_ (1.825. OOP)2 

(212)2  +  (5,862,500,000) 


65.1  , 


which  we  round  upward  to  n  *  66. 

The  allocation  to  strata  is  now  obtained  from  the  proportions  in 
column  F.  Thus,  stratum  1  will  have  a  sample  size  66(.ll)  -  7.3.  The 
results  for  the  five  strata  are 

7.3,  3.6,  21.7,  22.6,  10.8  . 


Since  these  values  are  not  integers,  it  will  be  safest  to  round  up  to 

8,  4,  22,  23,  11  , 

which  yields  a  total  sample  size  of  68. 

Because  Stanford  is  on  a  quarter  system,  the  sample  sizes  within  strata 
are  divided  into  thirds,  representing  the  three  quarters.  To  avoid  unequal 
numbers  per  quarter,  and  to  simplify  the  procedure,  we  round  upwards  so 
that  the  samples  are  divisible  by  3,  namely, 

9,  6,  24,  24,  12  . 

Thus,  for  example,  we  would  sample  4  per  quarter  in  the  fifth  stratum. 

The  total  sample  size  would  now  be  75. 

Remark.  An  alternative  procedure,  if  a  sample  is  not  divisible  by  3,  is  to 
randomize  the  remainder.  For  example,  in  the  fifth  stratum,  take  3  faculty 
per  quarter  and  assign  the  remaining  two  at  random  to  two  quarters. 
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4.  Derivations  and  Technical  Results 

We  now  provide  some  of  the  derivations  and  rationale  for  the 
formulas.  Formulas  that  are  standard  are  not  derived. 

Let  be  the  y  value  of  the  j-th  faculty  member  in  stratum 

1  for  quarter  (semester)  k.  (Since  Stanford  is  on  a  quarter  system,  we 
use  3  quarters  in  our  discussion.  However  the  procedures  are  easily 
modified  for  use  at  a  university  with  the  semester  system.] 

A  sample  of  size  n  is  chosen  and  then  partitioned  in  accordance 
with  the  allocation 


NiSi 

ni  "  —  ”  • 


“i<Ni  • 


in  each  stratum,  where  ve  assume  that  is  rounded  upward  to  be  divisible 

by  3  (or  2).  We  then  measure  the  y-value  of  1/3  (or  1/2)  of  the  sample  in  each 
stratum  each  quarter  (or  semester) . 

A  notation  is  needed  to  denote  measurements  taken  in  each  of  the 
quarters,  and  we  use  y^,  y^  and  y^  to  denote  the  sums  of  y-values  in 
quarters  1,  2,  3,  respectively.  That  is 

”i 

h  ‘  }ml  yi»  • 

I1 

*  ‘  jij+i  7«2 


y\  -  L 

1  j-2^+1 


rij3 


where  n^  ■  n^/3. 

The  sampling  procedure  has  two-stages:  In  the  first  stage  we  sample 
the  faculty  within  each  stratum,  and  in  the  second  stage,  sample  by 
quarters  (semesters) .  The  quantity  to  be  estimated  is 
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N, 


§  1  3 

5  l  l  l  Y  k/3  , 

1-1  j-1  k-1  X*K 


(3) 


where  g  is  the  nunber  of  strata.  The  estimate  used  Is 


**i^»i*»i*V  • 


<*) 


which  has  variance 

V(Y)-V(E(Y  |  sample))  +  E(V(?|  sample)) 


(5) 


Presumably,  V(E(Y| sample))  is  large  compared  to  E(V(Y| sample)). 
Consider  the  population  with  values 


yij  “  3(yijl  +  yil2  +  yij3)  * 


The  total  Y*  is  just  Y,  and  our  sanpling  scheme  reduces  to  a  single- 
stage,  stratified  simple  random  sample.  Also, 


N.  "i 


JL  A 

Y*  -  I  dr  l  . 

i  ni  j-i  13 


which  has  variance 


x-V- 

i  “i 


°i 

(1  -  Tp)  -  V(Y*> 
fti 


With  optimal  allocation,  n*  -  n(N. S*/T*) ,  where  T*  “  EN  S * .  Then 

1X1  Oi  Q 


V(Y*) 


.  m&2  (  sii\  v  T>,|is?  L  fi\ 
I n Vi V  »i  )'\  »  v  **/ 

T* 


l  Vi  -  *V  -  - v*  • 


T* 
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where  V*  -  ENjS*  .  Now,  S*  £  Sj,  so  that 


V(Y*)  £  -  V  . 

n 


Also,  because  the  expectation  is  linear. 


N.  “l 


V(E(Y |  sample))  -  V(J  ~  J  y*  )  -  V(Y*) 

1  ni  j-1  iJ 


Consequently  V(Y)^V(Y*),  and 


V(Y)  *  V(Y*)  £  —  V  , 


which  suggests  the  approximation 


—  -  V  *  V  (Y) 
n 


For  relative  error  e  with  confidence  coefficient  c,  we  must 


P{Y (1-e)  £  Z  £  Y(l+c)}  >  c  , 


where  Z  is  normally  distributed  with  mean  Y  and  variance  V(Y)  *  (Tz/n)-V. 
This  requires  that  the  condition 

2$(W)  =  ♦(cYA'V(Y))  -  1  >  c  (7) 


be  satisfied.  But  (7)  is  equivalent  to 


eY/ JvffJ  >  *_1(^)  , 


where  0  is  the  standard  normal  cumulative  distribution.  Consequently, 
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from  which  we  obtain 


(8) 


This  formula  is  slightly  optimistic  because  it  allows  for  fractional 

A 

allocation  to  strata  and  because  V(Y)  was  approximated.  Since  T  and 
V  are  not  known  exactly,  further  caution  whould  be  used  in  picking  n. 
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This  paper  describes  Che  background  of  the  Office  of  Management  Budget 
Circular  A-21  "Cost  Principle  for  Higher  Education"  that  describes  the  require¬ 
ment  for  effort  reporting.  A  sampling  procedure  Is  proposed  as  an  alternative 
to  1002  reporting. 
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